Mining the Data in Programming Assignments for Educational Research

In computer science and information technology education, instructors often use electronic tools to collect, compile, execute, and analyze student assignments. The assessment results produced by these tools provide a large body of data about student work habits, the quality of student work, and the areas where students are struggling. This paper reports on efforts to extract significantly more useful data from electronically collected assignments in computer programming courses. The work is being performed in the context of the most widely used open-source automated grading system: Web-CAT. We have enhanced a WebCAT plug-in to allow collection of data about the frequency and types of run-time errors produced by students, the frequency and types of test case failures that occur during grading, basic code size metrics, test coverage metrics, and more. This information can be combined with the results of “by-hand” grading activities to form a large, rich data corpus characterizing student behavior over many assignments in one course, over many courses, and even across semesters. The data collected in this way is a valuable resource for researchers in computer science education.